This tutorial will guide you on how to set up your dev environment to quickly start developing on Cortex XSOAR. While in Cortex XSOAR you can write code directly in the UI, which is awesome, you'll need a proper development environment external to Cortex XSOAR to contribute a full integration. This is because, in order to build a full fledged integration, you'll need to lint your code, run unit tests with pytest, create some documentation, submit your changes via git and more.
If you've been through this process already and just want a quick reference, you can jump to the Development Setup page, otherwise keep reading for more details.
This tutorial will guide you through the following steps:
Verify the requirementsFork the GitHub repoClone the GitHub fork locallySet up the environmentRun the linter and unit testsCreate a branch and integration directoryCommit and pushStep 1: Verify the requirements#Let's go make sure that all the requirements are satisfied, one by one.
Cortex XSOAR#We are assuming that Cortex XSOAR is already installed. For more information about installing Cortex XSOAR please refer to this article (Support Center credentials are required)
Check if your Cortex XSOAR License is correctly installed by navigating to Settings -> ABOUT -> License and make sure that everything is green:
PRO tip: you can quickly navigate to different pages within Cortex XSOAR by hitting Ctrl-K and then typing what you want. For the license page, for example, type /settings/license or just lic and select the autocompleted option:
GitHub#Not much to check here, just go to GitHub and make sure that you have an account or Sign Up for one:
Docker#Make sure that docker is installed on your system and is working correctly by running the hello-world container:
sb@dddd:~/demisto$ docker run --rm hello-worldHello from Docker!This message shows that your installation appears to be working correctly.[... output omitted for brevity ...]For more examples and ideas, visit: https://docs.docker.com/get-started/sb@dddd:~/demisto$infoMake sure that Allow the default Docker socket to be used (requires password) is enabled in Docker advanced settings.
tipIf you are using WSL2 on Windows, you can still use Docker Desktop from WSL.See the following tutorial for more details.
Great, all the prerequisites are set! We can get started.
Step 2: Fork the GitHub repo#Make sure you're logged on GitHub and navigate to the Cortex XSOAR Content Repo and click on Fork:
Once the fork is complete, copy the URL:
This is the fork where you will commit your code and, once ready, create the Pull Request to submit your contribution back to the Cortex XSOAR Content repository.
Step 3: Clone the GitHub fork locally#Back to the shell, create a folder (in the tutorial we'll use ~/demisto) and clone your fork of the content repository using git clone [your_fork_url], where [your_fork_url] is the URL you copied from GitHub in the previous step:
sb@dddd:~$ mkdir demistosb@dddd:~$ cd demistosb@dddd:~/demisto$ git clone https://github.com/[omitted]/content.gitCloning into 'content'...remote: Enumerating objects: 108, done.remote: Counting objects: 100% (108/108), done.remote: Compressing objects: 100% (90/90), done.remote: Total 101143 (delta 50), reused 53 (delta 18), pack-reused 101035Receiving objects: 100% (101143/101143), 110.65 MiB | 11.04 MiB/s, done.Resolving deltas: 100% (73634/73634), done.Checking out files: 100% (4522/4522), done.sb@dddd:~/demisto$Note: You must clone your fork of the repository, as you will need to be able to write into it. Do not clone demisto/content, as you won't be able to push commits.
Step 4: Setup environment#Option 1: Setup a remote environment#Follow the instructions in this guide.
Option 2: Setup a local environment#Let VSCode extension set up a local environment (Linux, MacOS, WSL2)#Follow this guide to set up a fully configured local environment.
Manually set up a local environment (Linux, MacOS, WSL2)#Operating System#We assume you have an operating system and that it is working. :)
Note: If you are using Windows with WSL, and your code resides in a shared folder on the Windows tree (i.e., /mnt/c/code/demisto), make sure that the folder is set to be case sensitive.
Python and pyenv#You will need python3 installed on your system. We recommend using pyenv. At the time of this writing, the latest version of Python 3.10 is 3.10.5.
Make sure pyenv in installed and that the eval "$(pyenv init -)" expression is placed in your shell configuration (~/.bashrc or ~/.zshrc) - more information about this.
sb@dddd:~/demisto$ eval "$(pyenv init -)"sb@dddd:~/demisto$ pyenv -vpyenv 1.2.15sb@dddd:~/demisto$~/demisto$If this doesn't work, follow the instructions here. Either Homebrew for MacOS or the automatic installer on Linux/WSL works fine.
Make sure that the required version of Python is available:
sb@dddd:~/demisto$ pyenv versions 3.10.5sb@dddd:~/demisto$If this doesn't work, follow the instructions here. Either Homebrew for MacOS or the automatic installer on Linux/WSL work fine.
If the required version of Python is missing, you will need to install it. As pyenv compiles CPython, you might need some libraries. Depending on your operating system, this article explains how to install the required dependencies and provides useful troubleshooting info.
Now is a good time to take a break since installing might take a while.
Install Python 3.10.5:
sb@dddd:~/demisto$ pyenv install 3.10.5Downloading Python-3.10.5.tar.xz...-> https://www.python.org/ftp/python/3.10.5/Python-3.10.5.tar.xzInstalling Python-3.10.5...Installed Python-3.10.5 to /home/sb/.pyenv/versions/3.10.5sb@dddd:~/demisto$ pyenv versions 3.10.5sb@dddd:~/demisto$And that's it! Again, if the installation fails, check out this page.
Poetry#Follow these instructions to install poetry
Node#Follow these instructions to install the nvm package manager.
After installing, run:
nvm install nodeBootstrap#Before running the bootstrap script that creates the virtual environment, let's set up pyenv to work correctly in the content folder you just cloned.
At the beginning, no local python interpreter has been set via pyenv:
sb@dddd:~/demisto$ cd contentsb@dddd:~/demisto/content$ pyenv localpyenv: no local version configured for this directoryYou can tell pyenv to use the latest version Python 3 you previously installed and verify that everything is set correctly:
sb@dddd:~/demisto/content$ pyenv local 3.10.5sb@dddd:~/demisto/content$ pyenv local3.10.5sb@dddd:~/demisto/content$ which python3/home/sb/.pyenv/shims/python3sb@dddd:~/demisto/content$ python3 -VPtyhon 3.10.5Now you can run the .hooks/bootstrap script that will install the dependencies and create the poetry environment:
sb@dddd:~/demisto/content$ .hooks/bootstrapInstalling 'pre-commit' hooks=======================Configure poetry to install virtual environment in the project repo (will be available in (.venv)Check if poetry files are validAll set!Installing dependencies...Detected local env.Installing dependencies from lock fileNo dependencies to install or update==========================Done setting up virtualenv with poetryActivate the venv by running: poetry shellDeactivate by running: deactivate=======================Running: npm install ...up to date, audited 230 packages in 1sNote: if you are using WSL and you see some errors about "python.exe" getting called, disable it in App Execution Alias (details).
Everything is configured, and you can start developing. When you work on your integration, you can activate poetry with the poetry shell command:
sb@dddd:~/demisto/content$ poetry shell(.venv) sb@dddd:~/demisto/content$Note the (.venv) in front of the prompt. You can always leave the poetry virtual environment using the deactivate command:
(.venv) sb@dddd:~/demisto/content$ deactivatesb@dddd:~/demisto/content$Step 5: Run the linter and unit tests#Our content ships with an HelloWorld integration that provides basic functionality and is useful to understand how to create integrations.
It's located in the Packs/HelloWorld/Integrations/HelloWorld folder. We will use demisto-sdk to run the linting and unit testing in order to make sure that everything is fine with the dev environment (python, docker, etc.).
First, make sure you are running inside the poetry virtual environment:
sb@dddd:~/demisto/content$ poetry shell(.venv) sb@dddd:~/demisto/content$Then, make sure that demisto-sdk has been installed automatically by the bootstrap script as part of the preqreuisites:
(venv) sb@dddd:~/demisto/content$ demisto-sdkUse demisto-sdk -h to see the available commands.Now, run the demisto-sdk lint command on the folder Packs/HelloWorld/Integrations/HelloWorld using the -i option,or if you want to run against all the committed files in your branch you can use demisto-sdk lint -g.It will run both the linters and pytest:
(venv) sb@dddd:~/demisto/content$ demisto-sdk lint -i Packs/HelloWorld/Integrations/HelloWorldExecute lint and test on 1/1 packagesHelloWorld - Facts - Using yaml file /home/sb/dev/demisto/content/Packs/HelloWorld/Integrations/HelloWorld/HelloWorld.ymlHelloWorld - Facts - Pulling docker images, can take up to 1-2 minutes if not exists locally HelloWorld - Facts - demisto/python3:3.8.2.6981 - Python 3.8HelloWorld - Facts - Tests foundHelloWorld - Facts - Lint file /home/sb/dev/demisto/content/Packs/HelloWorld/Integrations/HelloWorld/HelloWorld_test.pyHelloWorld - Facts - Lint file /home/sb/dev/demisto/content/Packs/HelloWorld/Integrations/HelloWorld/HelloWorld.pyHelloWorld - Flake8 - StartHelloWorld - Flake8 - Successfully finishedHelloWorld - Bandit - StartHelloWorld - Bandit - Successfully finishedHelloWorld - Mypy - StartHelloWorld - Mypy - Successfully finishedHelloWorld - Vulture - StartHelloWorld - Vulture - Successfully finishedHelloWorld - Flake8 - StartHelloWorld - Flake8 - Successfully finishedHelloWorld - Image create - Trying to pull existing image devtestdemisto/python3:3.8.2.6981-02b43abe979132c89892e089d5b8254dHelloWorld - Image create - Found existing image devtestdemisto/python3:3.8.2.6981-02b43abe979132c89892e089d5b8254dHelloWorld - Image create - Copy pack dir to image devtestdemisto/python3:3.8.2.6981-02b43abe979132c89892e089d5b8254dHelloWorld - Image create - Image sha256:ba9f6ede55 created successfullyHelloWorld - Pylint - Image sha256:ba9f6ede55 - StartHelloWorld - Pylint - Image sha256:ba9f6ede55 - exit-code: 0HelloWorld - Pylint - Image sha256:ba9f6ede55 - Successfully finishedHelloWorld - Pytest - Image sha256:ba9f6ede55 - Start============================= test session starts ==============================platform linux -- Python 3.8.2, pytest-5.0.1, py-1.8.1, pluggy-0.13.1rootdir: /devworkplugins: json-0.4.0, forked-1.1.3, mock-2.0.0, asyncio-0.10.0, datadir-ng-1.1.1, requests-mock-1.7.0, xdist-1.31.0collected 10 itemsHelloWorld_test.py ..........[100%]-------------- generated json report: /devwork/report_pytest.json --------------========================== 10 passed in 0.43 seconds ===========================HelloWorld - Pytest - Image sha256:ba9f6ede55 - exit-code: 0HelloWorld - Pytest - Image sha256:ba9f6ede55 - Successfully finishedFlake8- [PASS]Bandit- [PASS]Mypy - [PASS]Vulture - [PASS]Pytest- [PASS]Pylint- [PASS]Pwsh analyze - [SKIPPED]Pwsh test- [SKIPPED]Passed Unit-tests: - Package: HelloWorld - Image: demisto/python3:3.8.2.6981 - HelloWorld_test.py::test_say_hello - HelloWorld_test.py::test_start_scan - HelloWorld_test.py::test_status_scan - HelloWorld_test.py::test_scan_results - HelloWorld_test.py::test_search_alerts - HelloWorld_test.py::test_get_alert - HelloWorld_test.py::test_update_alert_status - HelloWorld_test.py::test_ip - HelloWorld_test.py::test_domain - HelloWorld_test.py::test_fetch_incidents######### Summary #########Packages: 1Packages PASS: 1Packages FAIL: 0Note that the tests run within a Docker container so, if everything worked well, it means that your development environment is up and running correctly!
Step 6: Create a branch#The Git Flow requires to create a branch with your new code, that you will later use to submit a Pull Request. This tutorial doesn't mean to be an exhaustive guide on how to use git: its purpose is just to make sure that you have all the requirements and tools in place to successfully develop a Cortex XSOAR Integration.
In order to create a branch, use the git checkout -b [branch_name] command, where the name of the branch corresponds to your integration:
(venv) sb@dddd:~/demisto/content$ git checkout -b my_integration_nameSwitched to a new branch 'my_integration_name'Step 7: Create your integration directory#Create a directory under Packs/ named after your product where you will put all your content files later, and add it to the staged changes in git. Make sure you use PascalCase in the directory name (i.e. MyIntegration).For a detailed description regarding what exactly a pack is please click here.
You can create a Pack and an Integration directory using the demisto-sdk init command.An example of creating a pack called MyNewPack, with an integration called MyIntegration, and with the metadata file created automatically:
➜ content-docs2 git:(add-pack-and-sdk-docs) ✗ demisto-sdk init --pack Please input the name of the initialized pack: MyNewPackSuccessfully created the pack test in: MyIntegrationDo you want to fill pack's metadata file? Y/N yDisplay name of the pack: MyNewPackDescription of the pack: A description for my newly created pack.Support type of the pack: [1] demisto[2] partner[3] developer[4] communityEnter option: 2Server min version: 5.0.0Author of the pack: Partner name The url of support, should represent your GitHub account (optional): https://github.com/The email in which you can be contacted in: partner@partner.comPack category options: [1] Analytics & SIEM[2] Utilities[3] Messaging[4] Endpoint[5] Network Security[6] Vulnerability Management[7] Case Management[8] Forensics & Malware Analysis[9] IT Services[10] Data Enrichment & Threat Intelligence[11] Authentication[12] Database[13] Deception[14] Email GatewayEnter option: 1Tags of the pack, comma separated values: Created pack metadata at path : MyNewPack/metadata.jsonDo you want to create an integration in the pack? Y/N yPlease input the name of the initialized integration: testDo you want to use the directory name as an ID for the integration? Y/N yFinished creating integration: MyNewPack/Integrations/test.Step 8: Commit and push#The last step is to commit your changes and push them to the origin in order to make sure that the pre-commit checks work fine.
But you can also run the hooks locally using the demisto-sdk, in order to do that you can run the commands:
demisto-sdk format - this will auto correct couple of things in order for our validation to pass.You can see the docs demisto-sdk validate -g - this will validate the integrity of the yml files, and will make sure they followour pre-set of roles. You can see the docs demisto-sdk pre-commit -i - this will run variety of checks and linters on yourchanged python files. You can see the docsFirst, run a git commit -m '[some commit message]', which will automatically run the pre validation checks:
(venv) sb@dddd:~/demisto/content$ git commit -m 'Initial commit of MyIntegration'Validating files...Starting validating files structureUsing gitRunning validation on branch my_integration_nameValidates only committed filesStarting validation against origin/masterThe files are validStarting secrets detectionFinished validating secrets, no secrets were found.Skipping running dev tasks (flake8, mypy, pylint, pytest). If you want to run this as part of the precommit hookset CONTENT_PRECOMMIT_RUN_DEV_TASKS=1. You can add the following line to ~/.zshrc:echo "export CONTENT_PRECOMMIT_RUN_DEV_TASKS=1" >> ~/.zshrcOr if you want to manually run dev tasks: ./Tests/scripts/pkg_dev_test_tasks.py -d Example: ./Tests/scripts/pkg_dev_test_tasks.py -d Scripts/ParseEmailFilesOn branch my_integration_nameUntracked files:.python-versionnothing added to commit but untracked files presentDon't worry about the .python-version file warning, that is generated by pyenv and shouldn't be added to the repository.
Note: since there are no files yet in the directory you have created (Integrations/MyIntegration in the example), it will not show up in your branch after the commit. Again, the purpose of this tutorial is just to make sure that all the components are in place.
If everything worked fine so far, now you can push to your branch with the command git push origin [branch_name]. You will be prompted for your GitHub credentials:
(venv) sb@dddd:~/demisto/content$ git push origin my_integration_nameUsername for 'https://github.com': [omitted]Password for 'https://[omitted]@github.com':Total 0 (delta 0), reused 0 (delta 0)remote:remote: Create a pull request for 'my_integration_name' on GitHub by visiting:remote: https://github.com/[omitted]/content/pull/new/my_integration_nameremote:To https://github.com/[omitted]/content * [new branch] my_integration_name -> my_integration_name(venv) sb@dddd:~/demisto/content$You can go back to GitHub and, under your fork, you should be able to see that there is a new branch with the name you provided (my_integration_name in this example):
Congratulations! You completed the set up of the Development Environment for Cortex XSOAR! Now you can start writing your code. Please have a look at the Code Conventions.
Thank for your time, we hope you enjoyed this tutorial. Please report issues and suggestions using the link below!